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It is argued that ultracold quantum degenerate gas of ytterbium ^^^Yb atoms having nuclear spin 1 = 5/2 
exhibits an enlarged SU(6) symmetry. Within the Landau Fermi liquid theory, stability criteria against Fermi 
liquid (Pomeranchuk) instabilities in the spin channel are considered. Focusing on the SU(n > 2) general- 
izations of ferromagnetism, it is shown within mean-field theory that the transition from the paramagnet to the 
itinerant ferromagnet is generically first order. On symmetry grounds, general SU(n) itinerant ferromagnetic 
ground states and their topological excitations are also discussed. These SU(n > 2) ferromagnets can become 
stable by increasing the scattering length using optical methods or in an optical lattice. However, in an optical 
lattice at current experimental temperatures, Mott states with different filling are expected to coexist in the same 
trap, as obtained from a calculation based on the SU(6) Hubbard model. 



I. INTRODUCTION 

Recently, the Kyoto group has managed to cool down to quantum degeneracy five Ytterbium isotopes fT\. The Ytterbium 
atom has a closed-shell electronic structure in the ground state ([Xe] 4f^^ 5s^ ^So), and hence its spin stems entirely from the 
nuclear spin, /. The case of the fermionic species ^^'^Yb is particularly interesting, as it has nuclear spin I = F = 5/2. Hence 
2F + 1 — n — 6, and the atom can be in six different internal states. At ultracold temperatures, experiments show that the 
scattering length is independent of the atom internal state |2 |. This can be understood from the absence of electronic spin in the 
atomic ground state, and the extremely weak dependence of the inter-atomic potential on the atomic nuclear spin. Thus, whereas 
for a spin-5/2 fermion the Lee- Yang-Huang pseudo-potential, depends on three scattering lengths |3|, af the previous 

observation implies that = = a'^ — Us- Mathematically, the interaction part of the Hamiltonian becomes: 

^int = ^ E K^o(*j) + alV^m + atV4^J)] S{r, - r,) = ^ 5(r, - r,), (D 

i<j = l i<j = l 

where M is the atom mass and Vpiij) the projector onto the state of total spin F for the pair of particles i and j. Therefore, the 
kinetic and interaction terms have the same symmetry, that is, the initial SU(2) spin-symmetry of the Hamiltonian describing 
an ultracold gas of ^^^Yb atoms is enlarged to an effective SU(6) symmetry. This is particularly interesting since enlarged 
symmetries usually lead to additional spectral degeneracies |4|, which in turn can lead to exotic (correlated) ground states and 
topological excitations |5 , 6, 7, 8|. 

The occurrence of SU(6) in an ultracold gas can also lead to new and interesting connections with high-energy physics, 
where SU(6) has been used to describe the flavor symmetry of spinful quarks, as nuclear forces seem to be spin independent 
to a first approximation |4|. Indeed, some of the phases discussed below can be regarded as (non-relativistic) pion condensates 
that spontaneously break SU(6). In addition, these phases also bear some resemblance to the quantum Hall ferromagnets El 
discussed in two-dimensional electron gases with valley symmetry (such as graphene). Their possible existence in ultracold 
gases of ^''^Yb can allow for larger control thanks to the large tunability of these atomic systems. In this regard, ultracold ^^-^Yb 
atoms in optical lattices may also allow the observation of other exotic time-reversal symmetry breaking phases such as the 
staggered flux phase |9|, which has been speculated as the explanation to the anomalous properties of the pseudo-gap phase of 
the high- Tc cuprate superconductors ifTol . 

In this paper, we study (in Sect.[ll|l the Fermi liquid instabilities in the spin SU(n = 6) channel of a strongly interacting ^^-^Yb 
gas. Focusing mainly on ferromagnetism, which breaks the SU(n) symmetry but not the space rotation invariance, we find in 
Sect. [Ill] that the paramagnetic to ferromagnetic transition to be generically first order for n > 2 at the mean-field level. On 
physical and symmetry grounds, we also identify the possible broken-symmetry ground states. The possibility of spontaneously 
breaking the SU(n) symmetry group in a cascade of phase transitions between different ferromagnetic phases hints at a much 
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richer phase diagram than in the spin- ^ case fT2l . These phases will also sustain exotic topological excitations, such as skyrmions 
in two dimensions and monopoles in three dimensions. As argued below, these phenomena may be observed by increasing the 
scattering length using an optical Feshbach resonance lfl4l or in perhaps also in a deep optical lattice. In Sect. IV we consider 
the situation in the lattice. Close to half-filling, i.e. 3 atoms per site, many phases, which may |61 or may not [9| break the 
SU(6) group, are likely to exist. However, at current accessible optical-lattice temperatures, atom hopping is largely incoherent, 
and Mott states are likely to coexist in the same harmonic trap. Indeed, for an SU(6) Hubbard model at high temperatures, we 
have computed the density profile showing the Mott plateaux (see Fig. [TJ. Finally, a summary of the results as well as a brief 
discussion of how to detect some of the phases discussed here can be found in Sect.jV] 



n. SU(n) FERMI LIQUID AND FERMI SURFACE INSTABILITIES 

We begin our analysis of the ^'''^Yb system, by exploring some consequences of SU(7i — 6) for the Fermi liquid phase of 
an interacting gas of ^^'^Yb atoms. Although we shall focus on the continuum case, many of there results in this section can 
be readily applied to the Fermi liquid phase of the gas loaded in an optical lattice (we neglect harmonic confinement for the 
moment; it will be considered briefly at the end, and more thoroughly elsewhere l,16J ). Following Landau L15J . we describe the 
low-lying excited states of the system using the distribution function n^(p) = (V'^(p)^/'"(p)) of a set of elementary excitations 
called Landau quasi-particles (QP, essentially atoms 'dressed' by the interactions). The latter are annihilated (created) by the 
Fermi operator ip"{p) {ipl^{p)) carrying (lattice) momentum hp and SU(n) index a = 1, . . . ,n. The excitation free energy of 
the QP states is given by the Landau functional (summation over repeated Greek-indices is implied henceforth): 

SF = Y, [e"{p) - ^^] SKip) + ^ E /,t(p, p')'5<(p)'5n^(p'), (2) 
p p,p' 

where il is the system volume, e"(p) is the excitation energy of a single Landau quasi-particle carrying momentum hp. We 
assume the ground state to be an SU(n) singlet and therefore the ground state quasi-particle distribution [n'^]^(p) = 9{fi — 

e°(p)) Sp, where /i is the chemical potential; (5n^(p) = "-^(p) — ['^'^]/3(p)- The Landau functions /"/(P;P') = /f-y"(P7P') 
describe interactions between quasi-particles. The expression for 6F can be considerably simplified with the help of group 
theory by noticing that 5ri^(p) transforms as a tensor belonging to the reducible representation of SU(n) n®n= 1 ® {n? — 1), 
where n and n are the fundamental and its complex conjugate representations, whereas 1 is the singlet and — \ the adjoint 
representations, respectively. Therefore, (5n^(p) — ^<5p(p)(S^ + ^"^^ (5m°(p) (T")^, where T° are the (traceless) generators 

of the SU(n) Lie-algebra obeying [T'^,T''] = i Ec=i^ ^"^^ '^"'^ choosing the normaUzation such that Tr (T^T'') = i(5"^ 
the structure constants A'^^'^ are fully anti-symmetric. In this representation, 5p{p) describes the total density fluctuations and 
5m°'{p) the SU(n) magnetization fluctuations. In addition, since the Landau functions transform as tensors belonging to n®n® 
n ® n = 1 © 1® non-singlet representations, all the tensor components are determined by just two scalar functions (compared 

to the five needed for a = 5/2 Fermi gas LU), that is, /"/(p,p') = F{p,p')5^5^s + 2/™(P,p') YTaJi C^")*- 

We next consider the stability of the Fermi surface (FS) of the SU(n) Fermi liquid just described above. For an isotropic 
FS, general stability conditions against FS deformations and pairing were obtained using the renormalization group by Chitov 
and Senechal [17J. They concluded that pairing occurs for attractive interactions. For repulsive interactions, d-wave pairing 
is also possible on a lattice near half-filling, but the pairing temperature rapidly decreases with increasing n ||6l. In the case 
of ^^■^Yb, s-wave interaction between atoms is naturally repulsive, as the scattering length is as = +10.55 nm f2\. This 
yields ppds — 0.1 at the center of the trap in current experimental conditions IT] |20l . Furthermore, currently accessible 
temperatures T//i ~ 0.4 ||T1 |20l are well above any pairing temperature scale. Therefore, the Fermi liquid phase should be a 
good starting description of the system. However, if the interaction is made sufficiently repulsive, the Fermi liquid can become 
unstable. The stability of the FS to the so-called Pomeranchuk instabilities can be assessed within Fermi liquid theory by 
considering the excitation energy of a quasi-particle distribution describing a deformation of the FS 115|. In matrix notation: 
n(p) = 9 [(/i — e"(p)) 1 + <5u(p)], where n(p) denotes the matrix whose components are n^(p), 1 is the unit matrix, and 
(5ii(p) is a matrix function that describes a small local deformation of the FS (p denotes those p-points lying on the FS). 
Expanding in powers of 5ii(p) = ^5up{p)l + 5m5^(p)T° up to second order, we obtain 5p{p) = S [fi — e"(p)) ^Up{p) + 

i<5' (Ai-eO(p)) U{Sup{p)f + ^ Ea('5"m(p))1 +••• and(5m"(p) = 5(Ai-eO(p))5<„(p) + --- In the continuum or in 
an optical lattice at low-filling, the FS is isotropic at the locus where |p | = pp, where pp is the Fermi momentum, and p/ |p | = p. 
For example, in three dimensions, we can expand dup{p) = J2lm ^''^p^ YLMip), and dup{p) = TIlm ^"^m^^^ Ylm{p), 
/P/m(p,p') = /P/m(p . p') = Y.LM /pAn^iV (p)>'lm (p'), whcrc Ylm{p) are the spherical harmonics. Therefore, 
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5F = 5Fp + 5Fm, where (note that Sup^^ , 6u'^^'^^ have units of energy) 
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2L + 1 



LA/ I 



(3) 
(4) 



We have introduced the (dimensionless) Landau parameters defined as JF£ — niV*'(/i)/^ and — N'^{jjl)f™, where = 
AI*pp / {2n'^h^) is the quasi-particle density of states (per species) at the FS in three dimensions, M* being the quasi-particle 
effective mass and hpp the Fermi momentum {pp — (Gtt'^po/"-)^^'^' where po is the total density). Hence, from Eqs. (U and 
the FS will be unstable if < -(2i + 1), for L = 0, 1, . . . 

The FS instabilities in the density channel {6Fp < 0, that is, < —{2L + 1)) are formally identical to those occurring 
in Fermi systems with no spin. They have received much attention recently |18|, and lead to phases where the rotation (or 
point-group, in the lattice) symmetry of the FS is broken {L > 0). On the other hand, much less attention has focussed on 
instabilities in the spin channel, which also break spin symmetry 1 19 1, and may occur in the interesting case of the ^^■^ Yb system 
with SU(n = 6) symmetry. Certainly, the most exotic states will be those resulting from an instability with L > in the spin 
channel, or, for a non-isotropic FS, one that breaks the lattice point-group besides SU(n). The resulting states have a much more 
complex order parameter, ^•f,^,/ cx / dp Ylm(p) ('T°)(^(5n^(pFP), that is the product of an orbital and an SU(n) part (similar 
to superfluidity in ^^He). 

However, as ultracold ^^-^Yb atoms naturally interact via repulsive s-wave (contact) interactions, in the isotropic case JF™ is 
expected to be the most negative Landau parameter, thus favoring SU(6) ferromagnetic correlations. Indeed, within Hartree- 
Fock theory, = No{p,)g{n - 1) > whereas J^^" = -No{p,)g < iNo{p) = Mpp / where M is the atom mass, 
g — Airfi^as/M. Hence, the SU(n) generalization of Stoner's criterion for ferromagnetism, F^ < —1, yields No{p,)g > 1, 
or equivalently, ppUg > ppa* — f , in the continuum case. It is worth noting that this criterion turns out to be the same 
as for the SU(2) case, that is, it is independent of n. The independence on n of the Stoner criterium in a SU(n) Fermi 
system can be understood using a simple energetic argument: To create a polarized ground state, imagine for example that 
SMo/{n — 1) fermions are removed from the Fermi surface of each of the rt — 1 flavors with a < n and added to the Fermi 
surface of the a = n flavor (so that the total particle number is unchanged). For small SMq, the kinetic energy of the system 

increases by {n — 1 ) ^^^^mv^r-r^ — \- 2nN°) ) ~ n "^-V^n ( V whereas the (Hartree-Fock) interaction energy decreases by 



(k-1)(k-2) / SMq 
2 I ri-l 



2nNo{ii) 

in - 1) 



' 2SlJVo(^i) 
\ n-1 



2(n-l)nA''o(At) ' 

ng(s Aio) fjgnce, upon comparing both energies, the dependence on n drops 



2(ri-l)Sl 

out and the system becomes unstable provided that No{jj,)g > 1, which is independent of n and agrees with the result obtained 
from Fermi liquid theory in the Hartree-Fock approximation. The cancellation of the dependence on n to the lowest order is 
a consequence of the fact that both the kinetic and exchage energies scale linearly with n (inspite of the fact that, naively, the 
interaction scales as n^). Nevertheless, as we shall see below, the nature of the transition from a paramagnet to an itinerant 
ferromagnet turns out to be very different for SU(n) with n > 2. 



III. SU(n) ITINERANT FERROMAGNETS 

The previous analysis using Landau Fermi liquid theory does not tell us anything about the order of the transition. In the sp in- 1 
(SU(2)) case, a Landau free-energy functional obtained from the microscopic Hamiltonian finds a continuous transition ifTTlllZl 
However, it has been recently pointed out that the coupling of the order parameter fluctuations to soft modes changes the order 
of the transition from second to first order at low temperatures lfT2l IT3l . In order to gain further insights into the nature of the 
transition at the mean field level, we shall derive in this section an effective action for the ferromagnetic order parameter starting 
from the microscopic model. To this end, we use the following operator identity for the interaction term of the Hamiltonian 
density: 



1 2 

(5) 



Hi„t(r) = -gc„(r)c;3(r)c'3(r)c"(r) = : [p{r)f : -5^ : [c„(r) (T'-^-^)^ c^^^ 

r=2 

In the above expression : . . . : stands for operator normal order, that is, the prescription that all atom creation fields, Ca{j^), 
should stand to the left of the destruction fields, c"(r). The matrices T'' — , ^ diag (1, 1, . . . , 1 — r, . . . , 0, 0) are the 

diagonal generators of the Lie algebra {i.e. the Cartan subalgebra). We next perform a Hubbard-Stratonovich decoupling of the 
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density (cx p ) and SU(n)-spin interaction terms, which yields the following action (/? = 1 /T, T being the absolute temperature) 



- \r^^'("'") + A^?(r,r)}. (6) 



Following the work by Hertz fTT| for the SU(2) case, we focus on the SU(n) spin fluctuations and therefore obtain an effective 
action for the fields Alr(r, r) by integrating out the Fermions and setting the density-fluctuation field (/3(r, r) = po (Po being 
the total density), that is, its saddle point value. Such a procedure yields the following effective action: 

S'offlM] = -Trln [-G^^l - gM] + drdrTrM^ir, r) (7) 

where M(r,T) = Y.r=2^r{r,T)T''''^^, such that TrM(r,T) = 0, and G^^{r - r',r - r') = 
— (jidr — fJ- + ^^7p^5Po — ~ r)(5(r — r'). However, it should be noticed that the present Hubbard-Stratonovich 

decoupling scheme using only the diagonal generators of SU(n) breaks the fuU SU(n) invariance of the theory. Yet, it does 
reproduce the correct Stoner criterion in the mean field (Hartree-Fock) approximation, which, as discussed above, comes 
out to be independent of n. The SU(n) invariance can be recovered by extending the functional integral over the entire set 
of traceless hermitian matrices M transforming according to the adjoint representation of SU(n). As noted in Sect. |llj a 
convenient of basis for this set is provided by the generators of the SU(7i) Lie algebra, T°, with a — I, . . . — 1. Hence, 
M(r, r) = rna{r, r) T", where mo(r, r) are real fields. Near the paramagnetic-ferromagnetic phase transition, we expect 
the order parameter to be small and therefore, we perform a series expansion in M neglecting its dependence in r and r. This 
yields the following (Landau) free-energy per unit volume: 

P P f^n^. p ^2 ^3 „4 

^ = ^ + y ^Tr M" = ^ + ^«2Tr M2 + ^v^Tv + ^v^Tr + ■ ■ ■ (8) 
It it ^-^ n it 2 3 4 

n=2 

The term of 0(.g) vanishes identically because M is traceless. However, for n > 2, terms of both even and odd order in 
g are non zero and occur in the free energy expansion in powers of M. This is to be contrasted with the SU(n = 2) case, 
where only terms of even order occur ifTTl [T2l ITSl . The coefficients vi = (g^^ + X2) and w„ = x„ for n > 2, where 



Xr: 



= ^Efc[Go(fcr = -J^^-^^n^^^ = (*en,k) andGo(fc) = (^e„ - e(p) + ^)-\ where e„ = f (n+i), 

*^(p) = "VjT"' ^'^'^ \\ss& shifted the chemical potential /i ^ /i — g(n — l)/npi^ to account for its renormalization due to 
interactions. 

We could have obtained the above free energy based on symmetry considerations of the order parameter. However, the micro- 

scopic approach allows us to relate the coefficients of the expansion to the model parameters. We next set M = X]a=i "^aT" 
in ([s]) and use the following SU(n) identity (see e.g. Il22l ). 

c—1 



where the group structure constants d'^'"^ are fully symmetric and /"'"^ fully anti-symmetric ll22ll . For n ~ 2, d"'"^ = but for 
n > 2 these structure constants are non-zero, which has important implications for the order of the paramagnetic-ferromagnetic 
phase transition. In terms of the ma components of the order parameter, the free-energy reads: 



a abc 



i d''''d'^'mamt,m,md }+■■■ (10) 

abcde 



The above expression shows explicitly that the Landau free energy contains a cubic term in the order parameter rua, which 
implies that, at least at the mean field level, the transition from the paramagnetic to the ferromagnetic phase is first order 
Thus, the system will exhibit hysteresis, and phase coexistence, with finite surface tension between the ferromagnetic and 
paramagnetic phase. Furthermore, the entropy will undergo a finite jump across the phase transition from the paramagnet to 
the SU(ri) ferromagnet. Moreover, the gas parameter resulting from Stoner's criterion pf^I = f , which is the point where the 
quadratic coefficient vanishes, is actually larger than the critical value of the gas parameter, that is, pj^a^ < ppat — f ■ The 
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latter corresponds to the point where both the paramagnetic minimum (M = 0) and ferromagnetic minimum (M ^ 0) have the 
same free energy. 

To illustrate the general ideas presented above, we shall next consider the case of the smaller group SU(3), which already 
contains essential ingredients of SU(ri > 2) ferromagnetism. The more complicated case of SU(6) relevant to an unpolarized 
mixture of ^'^■^Yb atoms will be studied elsewhere |16|. However, it is worth saying that the SU(3) case would correspond 
to an experiment where the system is prepared as a mixture containing an equal population of only three of the six internal 
states 1251 ||26l Considering a three dimensional gas in the continuum, setting M = L/^ (to3T'^ + mgT*) U , where U e SU(3), 

and using cyclic property of the trace along with the parametrization ma — '^^^"■.t) — ftiQ cos Q and mg = ^^"^"3 — rnp sin 6*, we 

V p P p 

arrive at the following expression for the (dimensionless) free-energy at T ^ fi: 

— ^1 ml - —TO^ sin 361+ —mf. - —m^ sin 361+ — m^(10 - cos 66*) + . .. 

Pfcls tt y 3 4 5 6 

In the above expression, the numerical coefficients are C2 = 47r, C3 = Stt/a/S, C4 = 167r^/3, C5 = 3207r^/27\/3, and cg = 
167r^/9. Using the above expression up to sixth order, we can also obtain the shift in the critical gas parameter (relative to the 
Stoner value, pi?a* = |): (pi?a^) — (pFa^)^^ — 0.066 or 1 — a^/a* ~ 0.094 ~ 10% at the the mean field level. Fluctuations are 
likely to decrease the critical value of the gas parameter even further from the Stoner value, and may also change the character 
of the transition (see e.g. |231|). In the SU(n > 2) case, fluctuations are responsible for the change of the order of the transition 
for SU(2). Thus, further analysis of the effect of fluctuations is needed but it is beyond the scope of the present work. 

For Os < the three energy has one minimum located at rTiQ = 0. However, for > Uc the free energy exhibits tree 
degenerate minima, corresponding (in 'cartesian' {m3,ms) coordinates) to Mq oc (0, —1) toq and Mq oc (±a/3/2, 1/2) ttiq. 
However, it needs to be noticed that these three minima represent the same physical state, as the result of the invariance of 
the free energy under the transformation 9—^9 + 2Trj/3, where j — 1,2, which corresponds to a cyclic permutation of the 
SU(3) indices 1 — > 2 ^ 3 — > 1, or in other words, to the existence of three (non-commuting) SU(2) subalgebras in SU(3), 
into which the larger group SU{3) can be spontaneously broken. Thus, let us choose Mq oc —mo = —mo (1 — 3e (g) e^^) 
(corresponding to = and mg = — mg), where e^^ ~ (0, 0, 1). This ferromagnetic state corresponds to a gas where one of 
the species Fermi surface {a = 3, in this case) grows at the expense of the two others, which remain degenerate. This state left 
invariant under the transformations generated by the SU(2) subalgebra span by {T^, T^, T"^}. Furthermore, it is left invariant 
by the U(l) transformations generated by T'^. Thus, the little group of transformations leaving the ground state invariant is 
H = SU(3) (8) U(l). If the interaction is increased further on, the remaining SU(2) group may be also spontaneously broken 
down to U(l) in a subsequent transition. 

Generally speaking, unlike SU(2) case, the SU(n) may be spontaneously broken in a cascade of phase transitions. A general 
analysis of the possibilities can be given by considering the structure of the order parameter As pointed out above, the order 
parameter is a traceless hermitian matrix, M — ™aTr"' which transforms according to the adjoint representation of SU(7i). 
Thus, when diagonalized, it has n — 1 independent eigenvalues. If only fc < n of them turn out to be equal, the symmetry 
breaking pattern (up to discrete groups) will be SU(n) SU(fc) x [U(l)]" Another more symmetric state occurs when there 
are only two distinct eigenvalues and hence SU(?i) SU(n — k) x lJ{k) (k < n/2). When all the n — 1 eigenvalues turn out 
to be different, SU(n) — [U(l)]" , etc. A simple example of the broken symmetry ground states (at the Hartree-Fock level) 
is provided by the state |<&(pp, . . . ,pP)) — Y[a=i n|p|<p° cj,(p)|0) , where |0) is the particle vacuum. The number of different 
eigenvalues tell us how many of the Fermi momenta coincide. More generally, any FM ground state can be considered to be 
adiabatically connected with an SU(n) rotation of |^'(Pp, ■ • ■ ,Pf))- When there are only two different eigenvalues, the order 
parameter manifold Ai — Gn.k — SlJ{n) /[SU{n — fc) x U(fc)], that is, a Grassmanian manifold |8|. In particular, for fc = 1, 
Gn^i — (DP"^^, the complex projective space. These manifolds have non-trivial second homotopy group, TT2{Gn.k) = 2^ 
(n > 2), which implies that these FM phases can sustain topologically stable excitations that are skyrmions in d ~ 2 and 
monopoles in d — 3. Furthermore, when SU(ri) breaks into a subgroup containing more than one U{1), TT2{Ai) = 'EP , where 
p < rt — 1 is the total number of [/(l)'s. The corresponding phases thus support complex types of topological defects described 
by several (integer) topological charges 

Finally, let us mention that as far as the experimental realization of SU(7i > 2) ferromagnetism is concerned, the above 
discussion suggest that the most convenient approach to observe an SU(n > 2) paramagnet to ferromagnet phase transition in 
the ^^-^Yb system is to increase the scattering length by means of an optical Feshbach resonance |14|. Ferromagnetism may 
also appear when the system is loaded in an optical lattice. However, this phase will compete with others (see next section) and 
further analysis will be required to understand the full phase diagram of the lattice system. 
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IV. "^YB GASES IN AN OPTICAL LATTICE: SU(n = 6) HUBBARD MODEL 



When the system is loaded in an optical lattice other phases may become more energetically favorable. In a uniformly filled 
lattice, as the filling ly = M / Ai (Af and M being the total atom and site numbers, respectively) approaches half-filling, — ^ 
other phases can become more favorable than ferromagnetism. Let us assume that the ^''■'Yb loaded in a lattice are accurately 
described by a sing le-band Hubbai'd model, = Ep eo(p) 4(p)c"(p) + | Er b(R•)]^ where eo(p) = -2t Et 

1 cos kiao 

is the free particle dispersion (oq is the lattice parameter), and p(R) = (R) the total site occupancy. This model will be 
accurate when the lattice is sufficiently deep. If the lattice depth is further increased so that hopping is suppressed along one 
direction, the system dimensionality will become effectively d = 2. In this case, there is strong evidence that for large values 
of n lElO, a staggered flux phase |10| with atom currents circulating in opposite directions in neighboring plaquettes will be 
favored as the ground state. This phase breaks time-reversal as well as lattice translation symmetries but does not break SU(n): 
thus the system will exhibit long-range order also at finite temperatures in d = 2. In Ref. |6| it was argued that n = 6 is indeed 
a borderline case where this phase competes with a flavor density wave that breaks both lattice translation symmetry and SU(7i) 
symmetry. The order parameter of this phase, _D^(Q) = ^ Ep('^/3(p)c"(P + Q)) (where Q — (tt, tt) at half-filling), is also 
a tensor belonging to the adjoint representaiton. Thus, if Tr []D(Q)T°] = D^(Q)(T'')^ ^ for any a = 1, . . . , n, the S\J{n) 
symmetry will be broken in one of the the same patterns as in the FM case. Thus, the ^^'^Yb gas in an optical lattice may be an 
ideal system to study the rich phase diagram resulting from the competition of all these phases. 

However, the temperatures that are currently achievable in an optical lattice (typically larger than the hopping amplitude t, 
see caption on Fig.[T]i are well above the temperature scales where the ordered phases discussed above may occur. Furthermore, 
the presence of the harmonic trap leads to an inhomogeneous filling of the lattice. The variation of the site occupation across the 
trap can be estimated in the so-called atomic {i.e. t — Q) limit of the SU(n) Hubbard model introduced above upon including the 

harmonic trap: iJat — ^ Er + ^ Er p(R-)' where Vt = \muj'^a'^ is the trapping energy and oq the (optical) 

lattice parameter The average site occupation can be thus obtained from (p(R)) = Tr [prc"'-'^'''^'^^-'/-'"] /Tre" 
with N = Er p(R-) ™d T the absolute temperature. Hence, 



-(Hat-pW)/T 



(p(R)) = r|^ln 



.p=0 



U ,.2 Vt(R/ao)^-H 



(11) 



where Cp — 



p!(n— p) 



is the energy degeneracy of a single-site state containing p particles. The chemical potential ^ must be 



adjusted to fix the total number of particles and T must be such that the entropy of the lattice equals that of the gas before 
adiabatically ramping up the lattice. A plot of the site occupancy as a function of the radial distance to the center of the trap |R| 
is displayed in Fig.[T] 



V. SUMMARY AND CONCLUSIONS 



To sum up, by using Fermi liquid theory, we have discussed Fermi liquid (Pomeranchuk) instabilities in the spin chanel of a 
strongly interacting ultracold ^^"^Yb gas exhibiting an enlarged SU(7i = 6) symmetry. Focusing on the Ferromagnetic instability, 
which does not break space rotation or translation symmetries, we have shown that the transition is generically first order (at 
least, at the mean level). Such an instability corresponds to a phase transition which can observed by increasing the scattering 
length using an optical Feshbach resonance or/and in an optical lattice. For the continuum case, the first order of the transition 
implies that the transition takes places at a slightly smaller value of the scattering length than the value provided by the Stoner 
criterion, which we find to be independent of the order of the group, n. Furthermore, using the smaller group SU(3) as an 
example, we have illustrated how the larger unitary symmetry is broken by an explicitly analysis of the Landau free energy 
derived from the microscopic Hamiltonian. Thus, we found that SU(3) is spontaneously broken down to SU(2)(8) U(l). 

On general symmetry grounds, we can expect a number of symmetry-breaking patterns for SU(6), which may be the result 
of not just one but a cascade of phase transitions between ferromagnetic phases. These SU(n) ferromagnets system can sustain 
exotic topologically stable excitations, such as skyrmions in d = 2 and monopoles in d = 3. The resulting phase diagram may 
be indeed quite rich, and will be explored elsewhere lfT6l . An interesting direction would be also to apply the analysis, based on 
Hertz theory [? ], to study other Fermi surface instabilities in SU(n) spin channel or to the flavor density wave of Ref. [6 | on the 
lattice. Based on the group theoretic properties of the order parameter, the latter may also turn out to be first order at the mean 
field level. Furthermore, in the optical lattice, the ^^-^ Yb system also offers other possibilites. such a realization of the staggered 
flux phase, which breaks the lattice but not SU(n) symmetry. However, under current experimental conditions, the temperature 
of the gas in the lattice is well above the ordering temperature for these phases. In this limit, we have obtained the density profile 
in a harmonic trap (see Fig.[T]i. 
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The fact that the spin of the Ytterbium atom in its ground state is entirely nuclear implies that its coupling to a real magnetic 
field is very weak and this renders magnetic fields impractical to detect the population of different species. However, the 
ferromagnetic phases and the topological defects discussed above could be detected by means of the optical Stern-Gerlach effect 
induced by off-resonant circularly polarized light |20|. Although this method has not been yet demonstrated experimentally, 
it provides the most dkect way to image the population of each species in a single shot measurement [20 J . Nevertheless, we 
hope that the possibilities discussed above for the observation of new and exotic many-body states in the ^'''^Yb will spur further 
theoretical and experimental research along these lines. The first step in this direction may be measuring the site occupation in 
an optical lattice, which can be carried out as explained in Ref.|2n 

We are indebted to Y. Takahasi and D. K. K. Lee for fruitful discussions. MAC acknowledges financial support of the Spanish 
MEC through grant No. FIS2007-06671 1-C02-02 and CSIC through grant No. PIE 200760/007. AFH acknowledges financial 
support from EPSRC(UK) through grant EP/D070082/1. 

Note added: After completion of our work, we became aware of the work by Gorshkov et al. |[24l . who also pointed out 
enlarged SU(n) symmetries of alkaline-earth atomic gases. 
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FIG. 1: Site occupation, p(lR|), as a function of the distance to the center of the cloud, |R|, in units of the lattice parameter ao, obtained from 
an SU(6) Hubbard model on a cubic lattice in d = 3. We have used parameters similar to those of current experiments 1 1 201 . In both the 
upper and lower panel U/t ~ 45 (lattice depth 10 Er). The upper panel shows the emergence of a half-filled (p(|R|) — 3) region near the 
center. The lower panel shows the emergence of the "Mott shell" structure as the lattice entropy per atom is reduced. The red curve corresponds 
to the currently achievable temperatures of the gas (before adiabatically ramping up the lattice) of Tinit /Tf ~ 0.4 fil l 201 (Tf being the Fermi 
temperature of the harmonically trapped gas). The entropy is estimated using (S'/A'^)iatt ~ tt^ (Tinit /Tf), for a trapped non-interacting gas. 
For the smallest value of the entropy, (S'/A'^)iatt = 1.86, the lattice temperature becomes comparable to the hopping (and therefore the atomic 
approximation breaks down). Hopping will further reduce the occupation near the edges of the cloud. 



